{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Cross-validation</h3>\n",
    "<p>In this notebook, we perform a cross-validation to try and choose the polynomial order for some synthetic data.</p>\n",
    "<p>The loss (in our case, this is equal to the squared error) on the training data will always decrease as we make the model more complex, and therefore cannot be used to select how complex the model ought to be. In some cases we may be lucky enough to have an independent test set for model validation but often we will not. In a cross-validation procedure (CV) we split the data into $K$ folds, and train $K$ different models, each with a different data fold removed. This removed fold is used for testing and the performance is averaged over the $K$ different folds.</p>\n",
    "<p>This process is illustrated in the following figure:</p>\n",
    "<img src=\"files/cvdiagram.png\">\n",
    "<p>We start by generating two datasets - one on which we will perform the CV, and a second independent test set.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "N = 100\n",
    "x = 10*np.random.rand(N,1) - 5\n",
    "t = 5*x**3 - x**2 + x + 150*np.random.randn(N,1)\n",
    "N_test = 200\n",
    "x_test = np.linspace(-5,5,N_test)[:,None]\n",
    "t_test = 5*x_test**3 - x_test**2 + x_test + 150*np.random.randn(N_test,1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>We now loop over model order (from 0th up to 10th order polynomials) and for each order, perform a $K=10$ fold CV. For each fold-order combintaion we compute the mean squared loss on the training data, the mean squared loss on the held out fold and the mean squared loss on the independent test set.</p>\n",
    "<p>Note that in many cases, it would be sensible to randomise the order of the data before placing it into the folds.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "max_order = 10\n",
    "X = []\n",
    "X_test = []\n",
    "K = 10\n",
    "sizes = np.tile(np.floor(N/10),(1,K))\n",
    "sizes[-1] = sizes[-1] + N - sizes.sum()\n",
    "c_sizes = np.hstack((0,np.cumsum(sizes)))\n",
    "X = np.ones_like(x)\n",
    "X_test = np.ones_like(x_test)\n",
    "cv_loss = np.zeros((K,max_order+1))\n",
    "ind_loss = np.zeros((K,max_order+1))\n",
    "train_loss = np.zeros((K,max_order+1))\n",
    "\n",
    "for k in range(max_order+1):\n",
    "    for fold in range(K):\n",
    "        X_fold = X[c_sizes[fold]:c_sizes[fold+1],:]\n",
    "        X_train = np.delete(X,np.arange(c_sizes[fold],c_sizes[fold+1],1),0)\n",
    "        t_fold = t[c_sizes[fold]:c_sizes[fold+1]]\n",
    "        t_train = np.delete(t,np.arange(c_sizes[fold],c_sizes[fold+1],1),0)\n",
    "        w = np.linalg.solve(np.dot(X_train.T,X_train),np.dot(X_train.T,t_train))\n",
    "        fold_pred = np.dot(X_fold,w)\n",
    "        cv_loss[fold,k] = ((fold_pred - t_fold)**2).mean()\n",
    "        ind_pred = np.dot(X_test,w)\n",
    "        ind_loss[fold,k] = ((ind_pred - t_test)**2).mean()\n",
    "        train_pred = np.dot(X_train,w)\n",
    "        train_loss[fold,k] = ((train_pred - t_train)**2).mean()\n",
    "    X = np.hstack((X,x**(k+1)))\n",
    "    X_test = np.hstack((X_test,x_test**(k+1)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Finally, we plot the mean performance in each case (where the mean is taken over folds).</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1056d6850>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEPCAYAAABlZDIgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcVOX+wPHPsArKjoCIgLu55pqQwrRoZlp6W9QK7ebt\n3m7LbftZajel8rZbt7plt7JcK61uuaRmWdjiXpnkjjsuICiiorKd3x/PGZiBAQaYYQbm+369zmvO\nPOecmWcQ58t5vs8CQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIUS8PAenAH/o+QCjwDbAHWA0E\nm50/BdgL7AKGmpX31V9nL/C6WbkvsEgv3wDE2f0TCCGEcLruqCDQDPBEBZH2wEvA4/o5TwAv6Ptd\nga2ANxAPZAAG/dgmYIC+vwIYpu/fB7yt748BPrH/xxBCCOFstwDvmz3/JyqQ7AIi9bIo/Tmou5Mn\nzM5fBQwEWgE7zcrHAu+YnXOFvu8FnLRT3YUQQtSShwNf+w9gMKqJyx8YDsSggkmWfk4W5cElGsg0\nuz4TaG2l/Khejv54RN8vBs7o7yeEEKKBeTnwtXcBL6LyJOdRzVklFc7R9E0IIUQj58iAAvCBvgH8\nC3WnkYVq6jqBas7K1o8fBdqYXRujn39U369YbromFjiG+ixBwKmKlWjfvr22b9+++n8aIYRwL/uA\nDrae7MgmL4AI/TEW+BPwEbAUmKCXTwC+1PeXovIjPkBboCMqGX8CyEflSgxACrDE7BrTa90CrLFW\niX379qFpmmyaxvTp051eB1fZ5GchPwv5WVS/oTpS2czRdyifAWFAEapH1hlUr67FwETgIHCbfu4O\nvXwHKh9yH+XNYfcBcwA/VC+vVXr5bGA+qttwLiogCSGEcAJHB5QkK2WngGurOP85favoF6CHlfJL\nlAckIYQQTuToJi/hYoxGo7Or4DLkZ1FOfhbl5GdRd4aaT2kSNL09UAghhI0MBgPUIk44uslLCLcT\nGhrK6dOnnV0NIWwWEhLCqVOVOsjWmtyhCGFnBoMB+X0TjUlVv7O1vUORHIoQQgi7kIAihBDCLiSg\nCCGEsAsJKEKIWhs+fDjz58+3+7m1kZaWRps2bWo+UTQY6eUlhJto0aKFKcnK+fPnadasGZ6engC8\n++67jBs3zubXWrFihUPOFY2b+wSUoiLw9nZ2LYRwmnPnzpXtt23bltmzZ3P11VdXOq+4uBgvL/f5\nahD24z5NXr/84uwaCOGS0tLSiImJ4aWXXqJVq1ZMnDiRvLw8RowYQUREBKGhoYwcOZKjR4+WXWM0\nGpk9ezYAc+bMYdCgQUyaNInQ0FDatWvHqlWr6nTugQMHSEpKIjAwkCFDhnD//feTkpJi0+fYuXMn\nRqORkJAQunfvzrJly8qOrVixgm7duhEYGEhMTAwzZ84EICcnhxEjRhASEkJYWBhJSUnS5bse3Ceg\nrF3r7BoI4bKysrI4ffo0hw8f5r///S+lpaVMnDiRw4cPc/jwYfz8/HjggQfKzjcYDGXNZwCbNm2i\nS5cu5Obm8vjjjzNx4sQ6nXv77bczcOBATp06RWpqKgsWLLC4tipFRUWMHDmSYcOGcfLkSd58803u\nuOMO9u7dC8DEiRN59913yc/PZ/v27WV3ZjNnzqRNmzbk5OSQnZ3N888/b9P7CevcJqBo36c5uwpC\nAGAw2GezJw8PD55++mm8vb1p1qwZoaGhjB49mmbNmtGiRQumTp3K2mr+KIuLi2PixIkYDAbGjx/P\n8ePHyc7OrtW5hw8fZsuWLTzzzDN4eXlx5ZVXcuONN9p0x7BhwwbOnz/P5MmT8fLy4qqrrmLEiBF8\n9NFHAPj4+LB9+3by8/MJCgqid+/eZeXHjx/n4MGDeHp6cuWVV9bhpydM3CaglP68DoqLnV0NIdA0\n+2z21LJlS3x8fMqeFxQU8Le//Y34+HiCgoJITk7mzJkzVX65R0VFle37+/sDljkbW849duwYoaGh\nNGvWrOy4rb24jh07VuncuLi4sma6zz//nBUrVhAfH4/RaGTDhg0ATJo0iQ4dOjB06FDat2/Piy++\naNP7CevcJqDkBcbCr786uxpCuKSKzTwzZ85kz549bNq0iTNnzrB27VrzRZccolWrVpw6dYoLFy6U\nlR0+fNima6Ojozly5IhF/Q4dOkRMjFrstV+/fnz55ZecPHmSUaNGcdttatWLFi1a8Morr7Bv3z6W\nLl3Kq6++ynfffWfHT+Ve3CagbPFPljyKEDY6d+4cfn5+BAUFcerUKZ5++mmHv2dcXBz9+vUjNTWV\noqIi1q9fz/Lly23KaVxxxRX4+/vz0ksvUVRURFpaGsuXL2fs2LEUFRWxcOFCzpw5g6enJwEBAWXd\npZcvX05GRgaaphEYGIinp2fZMVF7bhNQPssxoklAEcKqil/aDz/8MBcuXCA8PJzExESuv/76Kr/Y\nKybdrb2erecuXLiQ9evXExYWxlNPPcWYMWMsmuKqqrePjw/Lli1j5cqVtGzZkgceeID58+fTqVMn\nABYsWEDbtm0JCgri3XffZeHChQBkZGQwZMgQAgICSExM5P777yc5ObnK9xPVc5fuDFqvVln8cq4T\nnqdzQf4CEQ4ksw3bz5gxY+jatSvTp093dlWaNJltuJa6Xx1BfkBr2LrV2VURQlRhy5Yt7Nu3j9LS\nUlauXMnSpUsZNWqUs6slbOQ2ASUpCX5tkQxpac6uihCiCidOnOCqq64iICCARx55hHfeeYdevXo5\nu1rCRm7T5LVzp8Z/khbxn4ELYelSZ9dHNGHS5CUaG3s1eblNQCkt1ejR8gTbii7D41SO5FGEw0hA\nEY2N5FBqyWCALsYozjaPhG3bnF0dIYRochwdUKYA24F04CPAFwgFvgH2AKuB4Arn7wV2AUPNyvvq\nr7EXeN2s3BdYpJdvAOKqq0xSEmwNNkoeRQghHMCRASUeuAfoA/QAPIGxwGRUQOkErNGfA3QFxuiP\nw4C3Kb/VmgVMBDrq2zC9fCKQq5e9BlQ7b0JSEizNkwGOQgjhCI4MKPlAEeCPWnfFHzgG3AjM1c+Z\nC5j6BN4EfKxfcxDIAK4AWgEBwCb9vHlm15i/1ufANdVVqEcPWHE+mdK1P0BpaT0+mhBCiIocGVBO\nATOBw6hAkoe6M4kEsvRzsvTnANFAptn1mUBrK+VH9XL0xyP6fjFwBtWkZpWnJ3RIiua8Xzikp9ft\nUwkhysyZM4fBgwc7uxrCRThyWbb2wMOopq8zwKfAnRXO0fTN4VJTUwG4eBHm+HbiwbQ0kP7twg19\n9NFHvPrqq+zevZuAgAAuv/xynnzySY4cOcKUKVM4cOCAxfnFxcVER0czZ84chg8f7qRai4aQlpZG\nmovmmMcA75s9TwHeAnYCpvmrW6ES8KByKZPNzl+FavKK0q8xGYfKqZjOGajvewEnq6iLZrJhg6ZN\niV2gaaNHa0I4gvnvm6uZOXOmFhERoX3xxRdaQUGBVlxcrC1fvlx74okntIsXL2rBwcFaWlqaxTXL\nli3ToqKitJKSkkqv9+GHH2qDBg1qqOoLB6nqd5Za/sHvyCavXagvez9Ucv1aYAewDJignzMB+FLf\nX4pK2vsAbVGJ9k3ACVQ+5gr9dVKAJWbXmF7rFlSSv1p9+sDnOZJHEe7nzJkzTJ8+nbfffptRo0bh\n5+eHp6cnN9xwAy+88AK+vr7cdtttzJs3z+K6efPmcfvtt+PhUfPXxbp16+jfvz/BwcEMGDCA9evX\nlx2bM2cO7du3JzAwkHbt2pUtfpWRkUFycjLBwcG0bNmSsWPH2veDiybjccq7Dc8FvFE5jm+x3m14\nKioZvwu4zqzc1G04A3jDrNwXWEx5t+H4KuphEXWvvVbTzrVqr2nbtjVE8BdupuLvm6tYuXKl5uXl\nZfVOw+Tnn3/WAgMDtQsXLmiapml5eXman5+f9vvvv1s93/wOJTc3VwsODtYWLFiglZSUaB9//LEW\nEhKinTp1Sjt37pwWGBio7dmzR9M0TTtx4oS2fft2TdM0bezYsdpzzz2naZqmXbp0Sfv555/t9pmF\nbar6ncWF7lAAXgK6oboNT0D14DqFulvphBprkmd2/nNAB6AL8LVZ+S/6a3QA/mFWfgm4DXU3MxDV\nO6xGycnwR7hRug8L53DSGsC5ubmEh4dXe6eRmJhIZGQkX3zxBQCLFy+mc+fO9OzZs8bX/+qrr+jc\nuTN33HEHHh4ejB07li5durB06VIMBgMeHh6kp6dz4cIFIiMj6dq1K6Cmnj948CBHjx7Fx8eHxMTE\nWn824RrcZqS8uaQkWHleJooUTuKkNYDDwsLIycmhtIam3vHjx5c1e82fP5/x48fb9PrHjh0jNjbW\noiwuLo5jx47h7+/PokWLeOedd4iOjmbEiBHs3r0bgJdeeglN0xgwYADdu3fnww8/rPVnE6IhWdzG\nXbigaZ39Dmkl4S01rbS0AW4ohTup+PvmKvLy8rTmzZtrn332WbXnHThwQPP29tbWrVun+fj4aFlZ\nWVWea97kNX/+fG3AgAEWxxMSErS5c+dalF28eFF77LHHtMGDB1d6vZ9++klr1qyZtm/fPls/lrCD\nqn5ncbEmL5dRUFBQtt+sGUT0i+WiVwvYscOJtRKi4QQFBfHMM89w//33s2TJEgoKCigqKmLlypU8\n8cQTZefFx8czaNAgxo0bx9ChQ4mIiLDp9a+//nr27NnDxx9/THFxMYsWLWLXrl2MGDGC7OxslixZ\nwvnz5/H29qZ58+ZlS+1++umnZGaqoWbBwcFlzWOi8XGbf7UtW7ZYPE9Kgl0RMg2LcC+PPvoor776\nKjNmzCAiIoLY2FjefvttRo8ebXHehAkTOHLkSI3NXeZL+oaFhbF8+XJmzpxJeHg4r7zyCsuXLyc0\nNJTS0lJee+01WrduTVhYGD/++COzZqne/1u2bGHgwIEEBARw00038cYbbxAfH++Qzy8cy22mr3/+\n+eeZPLl8mMvq1bDlwblM7fUVLF7sxKqJpkamrxeNjUxfX0vm/eEBEhJgwZFktLVr65TgFEIIYclt\nAsq6dessInBAAAT0iOeShx/s2lXNlUIIIWzhNgHFz8+PjIwMi7KkJNgTJXkUIYSwB7cJKAkJCZWa\nvZKSYHWhUcajCCGEHbhNQElMTGTdunUWZYMGwbxDkkcRQgh7cKuAUvEOJSQEDO3aUqh5w969TqqZ\nEEI0DW4TUHr16sW+ffvIz8+3KE9KNpDRWqZhEUKI+nKbgOLj40OfPn3YuHGjRXlyMqwpNkpiXggh\n6sltAgpYT8wPHgxzDiSjpaVJHkWIaqSmppKSkuLsalQSHx/PmjU1LoXU6BiNRmbPnu3satSKWwUU\na4n5yEgoaNWeomID7NvnpJoJ0TDq8+VrqMOU+Q3BfPqX+rAlYMbHx/Pdd9/V+73mzJnD4MGDqz3H\nXp+rIblVQElISGDjxo2Vpu9OSjZwoI3kUUTT1xi/pFyJTKtTPbcKKBEREYSFhbFz506L8qQkSNNk\ngKNwL3PmzGHQoEFMmjSJ0NBQ2rVrx6pVq8qOHzhwgOTkZAIDAxk6dCg5OTkW12/YsIHExERCQkK4\n/PLLWWv2/8doNDJlyhSuuOIKgoKCGDVqFKdPn7b52mnTpjFo0CACAwO57rrryM3NLTs+f/584uLi\nCA8P57nnnrOok6ZpvPDCC3To0IHw8HDGjBlT9r4HDx7Ew8ODefPmERcXR8uWLcuuX7VqFc8//zyL\nFi0iICCA3r17V/p5paSkcPjwYUaOHElAQACvvPJKjZ/F2rLHu3bt4t5772X9+vUEBAQQGhpa47+V\npmnMmDGD+Ph4IiMjmTBhQlkHo4sXL3LnnXcSHh5OSEgIAwYMIDs7u8r3F/VXNr9/SkqK9u6771rM\n+X/okKYNDN2tlcbEyPooot5w0fVQNE3T4uPjtTVr1miaptYy8fb21t5//32ttLRUmzVrlhYdHV12\n7sCBA7XHHntMKyws1H744QctICBAS0lJ0TRN0zIzM7WwsDBt5cqVmqZp2jfffKOFhYVpOTk5mqZp\nWnJysta6dWtt+/bt2vnz57Wbb75Zu/POO22+tkOHDtrevXu1CxcuaEajUZs8ebKmaZq2fft2rUWL\nFtqPP/6oXbp0SXv00Uc1Ly+vss/073//W0tISNCOHj2qFRYWan/729+0cePGaZqm1nkxGAzaX//6\nV+3ixYva77//rvn6+mq7du3SNE3TUlNTyz6fLT+/mj5Ldcsez5kzp2wdmaoYjUZt9uzZmqZp2uzZ\ns7UOHTpoBw4c0M6dO6f96U9/KqvrO++8o40cOVK7cOGCVlpaqv36669afn5+te9fUVW/s8h6KNWz\nlpiPjYXjLTpScrEYDhxwUs2EuzA1O9V3s4e4uDgmTpyIwWBg/PjxHD9+nOzsbA4fPsyWLVt49tln\n8fb2ZvDgwYwcObLsugULFjB8+HCGDRsGwLXXXku/fv346quvyj7j+PHj6dq1K/7+/jz77LMsXryY\n0tJSm67985//TIcOHWjWrBm33XYbW7duBeCzzz5j5MiRDBo0CB8fH5599lmLtVP++9//MmPGDKKj\no/H29mb69Ol89tlnFs3c06dPx9fXl549e9KrVy9+//13QN0FaLVszqrus1S37HFt32fhwoU89thj\nxMfH07x5c55//nk++eQTSkpK8PHxITc3l71792IwGOjduzcBAQEAVb6/o7hdQLGWmAdINho42NYo\neRThcKYvrvpu9hAVFVW27+/vD8C5c+c4duwYISEh+Pn5lR2Pi4sre99Dhw7x6aefEhISUrb9/PPP\nnDhxouz8Nm3alO3HxsZSVFRETk6OTdea18vPz49z584BapnhmJgYizqHhYWVPT948CCjR48ue92u\nXbvi5eVFVlZWlZ/Z9Np1Ud1nqW7Z49o6fvw4cXFxZc9jY2MpLi4mOzublJQUrrvuOsaOHUvr1q15\n4oknKC4upnnz5nZ7f1u5XUDp3r07x48ft2iTBZVH+QHJowgB0KpVK06fPm2x0umhQ4fK7oxiY2NJ\nSUnh9OnTZdvZs2d5/PHHy84/fPiwxb63tzctW7a06dqqREdHc+TIkbLnBQUFFv+XY2NjWbVqlcVr\nFxQU0KpVqxpf25a7vorn1PRZhg4dyurVqzlx4gRdunThnnvusfm9zEVHR3Pw4MGy54cPH8bLy4vI\nyEi8vLyYNm0a27dvZ926dSxfvpx58+ZV+/6O4nYBxdPTk/79+7NhwwaL8qQkmHfYqMajCOHm4uLi\n6NevH9OnT6eoqIiffvqJ5cuXlx2/8847WbZsGatXr6akpISLFy+SlpbG0aNHAXUXtmDBAnbu3ElB\nQQHTpk3j1ltvxWAw1Hit6Xprbr75ZpYvX87PP/9MYWEh06ZNs2jOuvfee5k6dWpZMDt58iRLly61\n6TNHRUVx8ODBau/+IiMj2Wc2vKC6z1LdsseRkZFkZmZSVFRkU93GjRvHa6+9xsGDBzl37hxTp05l\n7NixeHh4kJaWRnp6OiUlJQQEBODt7Y2np2e17+8obhdQwHqzV4cOsJvOlBZcArO/BIRoqqzlYsyf\nf/TRR2zcuJHQ0FCeeeYZJkyYUHYsJiaGJUuW8Nxzz5UtJTxz5syyL2ODwUBKSgp33XUXrVq1orCw\nkDfeeMOmayvWw7ye3bp146233uL2228nOjqa0NBQi6a1hx56iBtvvJGhQ4cSGBhIQkICmzZtsvq6\nFd16662AWsq4X79+Vs+ZMmUKM2bMICQkhFdffbXaz1LdssfXXHMN3bp1IyoqioiIiCrrZHL33XeT\nkpJCUlIS7dq1w9/fnzfffBOAEydOcOuttxIUFETXrl0xGo2kpKRU+/6NVWfgN7PtDPAPIBT4BtgD\nrAaCza6ZAuwFdgFDzcr7Aun6sdfNyn2BRXr5BiCOyix6LqxYsUK76qqrKvVouO02TTsw4DZNmzOn\n2t4XQlSn4u+bOzLvoSRcX1W/s7hYL6/dQG996wsUAF8Ak1EBpROwRn8O0BUYoz8OA96mfD3jWcBE\noKO+DdPLJwK5etlrwIs1VWrgwIFs3ryZ4uJii/KkJPjJUwY4CmEPmgwAdDsN2eR1LZABHAFuBObq\n5XOBUfr+TcDHQBFwUD//CqAVEACY7l3nmV1j/lqfA9fUVJGQkBBiY2NJT0+3KE9OhoVHjZKYF8IO\nZES++/FqwPcaiwoWAJGAqR9flv4cIBrVbGWSCbRGBZhMs/Kjejn6o6nbRzGqWS0UOFVdZRISEli3\nbp3FiNiuXWHT2cso8TiH5+HDaoCKEKLWvv/+e2dXQThBQwUUH2Ak8ISVY7Vup6uL1NTUsn2j0Uhi\nYiJr1qzh/vvvLyv38IDBSQaOnUimzdq14IIzqwohhKOkpaWRVo8m/4YKKNcDvwAn9edZQBRwAtWc\nla2XHwXamF0Xg7ozOarvVyw3XRMLHEN9niCs3J2YBxRQ3fZmzJhRqaJJSbD+y2TapKVJQBFCuBWj\n0YjRaCx7/vTTT9fq+obKoYyjvLkLYClg6oM4AfjSrHws6o6mLSrRvgkVePJR+RQDkAIssfJat6CS\n/DXq3LkzeXl5FqNzQQWUj48bJY8ihBAO8DDqr34DMBvV/fe6WlzfHMhBJdVNQoFvsd5teCoqGb+r\nwvuYug1nAG+YlfsCiynvNhxvpQ5Wu8QNHz5c+9///mdRVlSkaUEBJVpJaJimHTli7955wg2EhISY\nmnFlk61RbCEhIVZ/l/XjNrOlG8Y2oCfqy/1e4ClgPqorcGOh/2wszZgxgzNnzvDyyy9blA8bBh/k\n/YnoB2+GO+5oqDoKIYRL0Xvq2dxdz5YmL9OL3YAKJH/UvlquqcqJIpNho59RxqMIIUQt2BJQfkE1\nSw0HvgYCgdJqr2gkBgwYwNatW7l06ZJFeVISLM5KljyKEELUgi0B5W7UdCj9gPOAN/BnR1aqobRo\n0YJOnTrx22+/WZT36wfLD/WgNCcXjh1zUu2EEKJxsSWgJKCmUMlD9a76J2rwYJOQmJhYacEtX1/o\nN8CDk10Gy12KEELYyJaA8g7qzqQX8Ciql9U8R1aqIZlGzFeUlASb/Y0SUIQQwka2BJRiVNexUcBb\n+hZQ7RWNiCkxX7EXWFISfHYyWRLzQghhI1sCylnU2JA7geWAJyqP0iS0bduW4uJii1XgAAYOhP9l\n9ETLyoIKgx+FEEJUZktAGQNcQiXnT6AmY3y52isaEYPBYLX7cPPm0L2XJ7mXSR5FCCFsYUtAOQ4s\nRI1mHwFcpAnlUMB6Yh5Us9cvAUYJKEIIYQNbAsptwEbgVn1/k77fZFSXmP/ilORRhBDCFrZOvXIt\n5TMCt0RNwNjTUZVyAKtTr5hcuHCB8PBwTp48ib+/f1n5mTMQ27qEPK8wDHv2gA1rPwshRFPhqKlX\nTpo9z63NGzQGfn5+dO/enS1btliUBwVB+06enO42SJq9hBCiBrYElFWoKVfuQo2QXwGsdGCdnKKq\neb2SkmBrkFECihBC1MCWgPI48F/UwMYe+v7jjqyUM1SVmE9OhiV5kkcRQoiaNKmmq2pUm0MByMzM\npHfv3mRnZ5vaDQE4eRK6dCgmhzAMGRnQsqWj6yqEEC7BnjmUc6hBjda2/LpX0TXFxMTg5+dHRkaG\nRXnLlhAV40V+jyvhhx+cVDshhHB91QWUFqgpVqxtgY6vWsNLSEiocjxKeohMZy+EENVpqDXlG4Xq\nEvPLzholoAghRDUkoJipKjE/eDDMTe+DduAA5OY6oWZCCOH6JKCY6dWrF/v27SM/3zJFFBMDzYO9\nOdczEX780Um1E0II1yYBxYyPjw99+vRh48aNlY4lJ8P2MOk+LIQQVZFeXhVUl5hfUWCUPIoQQlTB\nll5erwNPoKatb40a1Pi646vmHNUl5udu74eWkQGnTzuhZkII4dpsafK6EXgbdVeSD8wCbqrFewQD\nnwE7gR3AFUAo8A2wB1itn2MyBdgL7AKGmpX3BdL1Y+YBzRdYpJdvAOJqUbdKEhIS2LhxI6WlpRbl\nbdtCiYc3F3olSB5FCCGssCWgnEet1uipb3egmsNs9Tpq/q/LUDMU7wImowJKJ9TMxZP1c7uiFvTq\nCgxDBTLTKM1ZwESgo74N08snoias7Ai8BrxYi7pVEhERQVhYGDt37rQoNxjUXcrOCMmjCCGENbYE\nlNtR66Bk6dttepktgoDBwAf682LgDOquZ65eNhe1Xj2oO5+PgSLgIJCBuqNphWp+26SfN8/sGvPX\n+hy4xsa6Vam6Zq+VF42SRxFCCCtsCSgHUF/a4fp2E+rL3hZtUVPffwj8CrwHNAciUcEJ/TFS348G\nMs2uz0TlbSqWH9XL0R9NC8KbAlaojfWzqqrEfHIyzN/ZH/bsgby8+ryFEEI0OV42nNMZ1fQUBXRD\nNVvdCMyw8fX7AA8Am4F/U968ZaLpm0OlpqaW7RuNRoxGY5XnJiYm8vrrlfsddOkCp875cLHnAJr9\n9BOMGOGAmgohhHOkpaWRVo8mfVtmkfwBmAS8A/TWr/kDFVxqEgWsR92pAAxCJd3bAVcBJ1DNWd8D\nXSgPNi/oj6uA6cAh/ZzL9PJxQBLwd/2cVFRC3gs4jlpV0lyNsw2bKykpITQ0lP379xMWFmZx7Oab\nYbrHs/SMOwOvvGLzawohRGPjiBUb/VFryptoqByHLU6gmqM66c+vBbYDy4AJetkE4Et9fykwFvBB\nBaGOqLzJCVQPsytQHy4FWGJ2jem1bkEl+evF09OT/v37s2HDhkrHkpJg9SWZKFIIISqyJaCcBDqY\nPb8FdRdgqweBhcDvqOayf6HuQIagug1fTfkdyQ5gsf64EriP8uaw+4D3Ud2DM1B3JgCzgTC9/GEq\nN6nVSXWJ+QV7BsDOnZDfZMd3CiFErdlyK9MeeBdIAPJQSfo7sD0x7wpq1eQFsHLlSl5++WW+++47\ni/KSEggLg+xuV+Hz5CQYPtye9RRCCJdh7yYvT1Se4hogApXnuJLGFUzqZODAgWzevJni4mKLck9P\nuPJKyIgxSrOXEEKYqSmglKAS6QbUYEa3aeMJCQkhNjaW9PT0SseSk2FNsQxwFEIIc7bkULaiEuAp\nwM369idzlwMxAAAgAElEQVRHVspVJCQkVJ1HyRgI27fD2bNOqJkQQrgeWwJKM+AUKnk+Qt9GOrJS\nrqKqxHyfPrBjfzOKe/WFn392Qs2EEML12JxsaeRqnZQH2LlzJzfccAP79++vdOzaa+HtiOl0iiuE\n55+3Rx2FEMKl1DYpb8tIeT/UBIxd9X3TN/Pdta1cY9O5c2fy8vI4ceIEUVFRFseSkuD77cl0SnvS\nSbUTQgjXYkuT13zUXFvDgDSgDbWbbbjR8vDwqHbBrY/2D4T0dDjnFj8OIYSoli0BpQPwFCqIzAWG\no0asu4WqEvNXXAG/7PSnpGdvsHJcCCHcjS0BpVB/PAP0QC2GVXGurCarqsS8nx/07g2H2hllPIoQ\nQmBbQHkPNR38P1HzZu0AXnJkpVzJgAED2Lp1K5cuXap0LCkJfkDGowghBNgeUE4Ba1ETNrZEzTzs\nFlq0aEGnTp347bffKh1LSoJPDiXA1q1QUOCE2gkhhOuwpZfXdLN98763z9i5Li4rMTGR9evXM3Dg\nwArlcPNvzSnteTke69fDNfVeLFIIIRotW9eUP6dvpaikfLwD6+RyqkrMBwTAZZdBZgdp9hJCiLoM\nbPQFVgPJdq6LI9VpYKPJ/v37GTx4MJmZmaaBPmUeewz65q7m9v0z4Icf6ltPIYRwGY5YYKui5pSv\n5+4W2rZtS3FxMUeOHKl0LDkZFh1JhF9/hQsXnFA7IYRwDbYElHSzbTuwG6i84HoTZjAYquw+PGgQ\nfL+5BVr3HmBlhUchhHAXtgSUkWbbdUA08KYjK+WKTIn5ikJDIT4eTnSWPIoQwr3ZElDyzbYCIAA1\nLsW0uYWqEvOgug+v8zHKAEchhFuzJdlyEIgFTuvPQ4DDqC7EGtDOITWzr3ol5QEuXLhAeHg4J0+e\nxN/f3+LY4sXw+ZyzLPqhFeTkQLNm9XovIYRwBY5Iyn+DWgMlTN9uQPXyakvjCCZ24efnR/fu3dmy\nZUulY0lJ8M2GALRu3WDjRifUTgghnM+WgJIArDB7vhJIdEx1XFtVifmoKGjZEk52lTyKEMJ92RJQ\njqHm8YpH3ZU8CRx1YJ1cVlWJeVB3KRubGSWPIoRwW7YElHFABPAF8D99f5wjK+WqTIl5a/mYpCT4\n7MQg2LQJrEwkKYQQTZ0tASUX+AfQG+gHTENNFmmrg8A24Ddgk14WisrN7EHlY4LNzp8C7AV2AUPN\nyvuixsLsxXIcjC+wSC/fAMTVom61EhMTg5+fHxkZGZWOJSXB1+sD0bp0UUFFCCHcjC0B5WMgEDVC\nPh01ff3jtXgPDTCiAtIAvWwyKqB0Atboz0EtMzxGfxwGvE15D4NZqKWIO+rbML18IirodQReA16s\nRd1qraoVHOPiwNcXTvc0SrOXEMIt2RJQuqLGoIxCJeTjgZRavk/Fbmc3olZ/RH8cpe/fhApgRag7\nmwzU6pCtUONfTH/6zzO7xvy1PgccOuVvVYl5UHcpm/0lMS+EcE+2BBQvwBv1Bb4M9WVfm0EdGvAt\nsAW4Ry+LBLL0/Sz9OahR+Jlm12ai5g2rWH6U8vnEWgOmSbaKUStLOmzAZU2J+f+dHKy6DhcWWj1H\nCCGaKlvWQ/kv5XmQH1B3KGdq8R5XAsdRC3N9g8qNmDMNkHSo1NTUsn2j0YjRaKzT6/Tq1Yt9+/aR\nn59PYGCgxbHkZJgxIxg6doTNm+HKK+tRYyGEaFhpaWmk1aOFpS7T1xsAT9TdQG1NR62rcg8qr3IC\n1Zz1PdCF8lzKC/rjKv2aQ/o5l+nl44Ak4O/6OamohLwX5cHLXL1HyptLSkriqaeeYsiQIRXeBFq1\ngj0jHiWwXThMnWq39xRCiIbWENPXa9geTPxRuQ9QSf2hqMT+UmCCXj4B+FLfXwqMBXxQY146ovIm\nJ1B5nCtQHy4FWGJ2jem1bkEl+R2qqsS8waCavX5pIXkUIYT7qUtAqY1I4EdgK7ARWI7qJvwCMATV\nbfhqyu9IdgCL9ceVwH2UN4fdB7yP6h6cgbozAZiNmhJmL/Aw5Xc5DlNTYv7L3MFqKvuiIkdXRQgh\nXEZdmrwaI7s2eWVnZ9O5c2dyc3Px8LCMydu2wS23wB7/y2HWLEhIsNv7CiFEQ3JUk9eVwB2opqUJ\nwPha16wJiYiIICwsjJ07d1Y61r27mnD4fH+jjEcRQrgVWwLKAuBlVFDpp2/9HVmpxqCqZi8PD7WK\n49YgyaMIIdyLLd2G+6IGNzq8a29jYkrM33PPPZWOJSfD0p1JXLlugsqjeHs7oYZCCNGwbLlD+QPV\ntVeYqSkxv3JTmFob+NdfG7ZiQgjhJLYElJaoXlerUSPll6G66rq17t27c/z4cXJzcysd690bDh6E\niwONkkcRQrgNWwJKKmraleeAmfr2qgPr1Ch4enrSv39/NmzYUOmYl5fq3LUtJFkCihDCbdgSUNKq\n2NxeTc1ey/OT4KefoLgukwoIIUTjYusSwJtRU6YUAaWoUetur6oR86DnUba0hDZtYOvWBq6ZFZoG\n2dnqUQghHMCWgPIf4HbUSPRmqPVH3nZkpRqLgQMHsnnzZoqt3IEMGAA7d0JhghO6D587p0bqv/su\nPPggGI0QFgaxsTBpUsPWRQjhNmwd2LgXNSFkCfAh5YtbubWQkBBiY2NJT0+vdMzXF/r1gx0RRsfl\nUYqLYdcuWLwYnnoKRo2Cdu0gIgIeeADWr4e2beGf/1TR7cgRWLBAlQshhJ3ZMg7lPGqZ3d+Bl1AT\nNbrLlC01Mq0z37t370rHkpJgxZkkLv/xHigpAU/Pur2JpkFWlprXJT1dbdu2qWASHQ09eqjtzjvV\nY4cOVb/XG2/A3XfDb79Bs2Z1q48QQlhhS2CIRy2C5QM8gloO+G3UBI2NhV3n8jL3wQcfsGbNGhYu\nXFjp2LffwtNPw4+5XdWdQZ8+Nb/g+fOwfbtl4EhPh9JS6NlTBQzTY7du0KJF7SqsaWqysY4d4YUX\naj5fCOG2ajuXl60n+gNtgN11qJMrcFhA2blzJzfccAP79++vdOz8eYiMhLxxf8frso7w6KPlB0tK\nYN++yoHj6FHo3NkycPTooRZaMdjpxjArS7328uXQ3+1n0RFCVMERAeVG1Fxevqi7ld7A03p5Y+Gw\ngFJaWkp4eDg7duwgKiqq0vGBA2H20EV0++5NdWdgChw7dqhcR8XA0bFjw0zVsnAhPP88/PKLSvgI\nIUQFjggov6LWLPkeFUxATcfSvbaVcyKHBRSAG264gb/85S+MHj260rHHH4eWhhwmbUuB9u3LA0f3\n7lBhCeEGpWlw003Qqxc8+6zz6iGEcFm1DSi2JOWLgLwKZaW1qFOTZ0rMWwsoycnw2mvhTPp2pRNq\nVg2DAd55By6/HP70JzVfjBBC1IMt3Ya3o9ZC8UItyfsmYH14uJuqbsT8lVfCxo1QWNjAlbJFdDS8\n9BL8+c8uWkEhRGNiS0B5EOgGXAI+Ro2Sf9iRlWpsBgwYwNatW7l06VKlY8HBqhevy046PGGCCizS\n40sIUU+2BJTzwFTKF9d6ErjoyEo1Ni1atKBTp0789ttvVo8nJbnwHJEGA/z3v/Dmm6qzgBBC1FF1\nOZRlqEW1rCVkNBpXLy+HS0xMZP369QwcOLDSsaQkeO01labw8FBjDj08LPetldmyb8u5HjX92dCm\nDTz3nGr62rBBTZcshBC1VF32/iSQiWrm2ljhfA1w1b+5rXFoLy+ABQsWsGTJEj799NNKx06fVi1L\nFy+q4SelpWqrab8251a3r2kqqHh5wcsvwz/+YfUnBEOHwjXXwOTJDv1ZCSEaB3t2G/YChgDjgB7A\nV6jgsr0e9XMWhweU/fv3M3jwYDIzM03/CC5D01RgOXhQrdPy1VdVjGc8eFBNQPbjj3DZZQ1cSyGE\nq6ltQKmuMaQYWAmMBwaiplpZCzxQj/o1WW3btqW4uJgjR444uyqVGAyqCax9e5g1C8aOhTNnrJwY\nHw/PPKPm+iopaehqCiEauZpa15sBNwMLgPuB14EvavkensBvqJwMQCjwDbAHtaxwsNm5U1AzG+8C\nhpqV9wXS9WOvm5X7Aov08g1AXC3rZjcGg6Ha7sOu4uab4brr4G9/q2JplHvvVSPn//3vBq+bEKJx\nqy6gzEeNN+kNPAP0B54FjtbyPR5CrUlv+vqajAoonYA1+nOArsAY/XEYagJK063WLNQ6LB31zTR9\n/kQgVy97DXixlnWzK1Ni3tXNnKlms58928pBDw94/301LcuePQ1eNyFE41VdQLkD9UX9ECqwnDXb\nbF2xMQYYDrxPeXC4EZir789FrVcPcBMqR1MEHEQ1sV0BtAICgE36efPMrjF/rc+Ba2ysl0OYRsy7\nOj8/+OQTmDJFTWxcSYcOag2ViRNV8kUIIWxQXUDxQH2RW9tsnYTqNWASllO1RKKmw0d/jNT3o1G9\nykwygdZWyo/q5eiPpqRFMXAG1aTmFH379mXHjh0UFBQ4qwo2u+wyNUh+zBiwWt0HH1TB5D//afC6\nCSEaJ0cOOBgBZKPyJ8YqztEobwpzqNTU1LJ9o9GI0Wi0+3v4+fnRvXt3tmzZQlJSkt1f397uugvW\nrIFHHlFjGy14esIHH6i5Y264QWX0hRBNWlpaGmn1WLLckf1bnwNSUHcOzVB3Nf9D5WKMqJUfW6Fm\nMe5CeS7FNAfIKmA6cEg/x9SPdRyQBPxdPycVlZD3Ao4DLa3UxeHdhk0eeeQRIiMjmdxIxnKcPavW\n/frXv+C226yc8PLLsGKFijw1jpAUQjQl9uw2XF9TUYtytQXGAt+hAsxSYIJ+zgTgS31/qX6ej35N\nR1Te5AQqZ3MF6oOlAEvMrjG91i2oJL9TNZbEvElAgMqnPPAAWFkjTC0KVlBg5RZGCCEsNdQIvGTg\nMVQSPRRYDMSiku+3UT49/lTgbtRdzUPA13p5X2AO4AesAExjvX1RvdF6o3p7jdVfs6IGu0PJzMyk\nd+/eZGdnu9wAx+q8/rpac+unn8DHp8LBHTvU/DG//AJxTuuZLYRoYI5aArixa7CAAhAbG8uaNWvo\n2LFjg71nfZnW2+rcWbVyVfLcc5CWBl9/bb+liIUQLs2VmrzcVkJCQqNq9gIVIz78EBYtgpXW1gKb\nNAlyc1WiXgghrJCA4gCNYcS8NWFhsGCBmnT42LEKB729VcSZPBkyM61eL4RwbxJQHKCxJebNJSXB\nfffBnXdamc6rZ0+Vva9y3hYhhDtzl8bwBs2hFBYWEhoayrFjxwgMtHUMqOsoKYFrr4Wrr4annqpw\nsLBQTVX82GMwfrxT6ieEaBiSQ3EBPj4+9OnTh40bN9Z8sgvy9FQ9vt5+W81kb8HHRzV9/d//wfHj\nTqmfEMI1SUBxkMbc7AVqmfnZs+GOO1Qu3kKfPvDXv8Lf/y5NX0KIMhJQHKSxTBRZneHD1ej5P//Z\nStx46inIyFCjIoUQAsmhOEx2djadO3cmNzcXj0Y8ZUlhIQwapO5UHnqowsHNm2HECNi2DSIjrV4v\nhGi8JIfiIiIiIggLC2Pnzp3Orkq9+Piom5AZM9RAeQv9+6sZJh+QRTyFEBJQHKqxjkepqF07NYv9\n2LFqMkkLqanqDuWzz5xRNSGEI1y4oNYLryUJKA7U2BPz5saMgauuspKH9/NTvb4efBBycpxWPyGE\nHeTnw4svqr8iV62q9eUSUByoKSTmzf3737B1K8ydW+FAYqK6ffnHP6xeJ4RwcTk5MG2aWvdo2zZY\nvRqWLKn5ugokoDhQ9+7dOX78OLmV+t02Tv7+aq6vSZPUmvQW/vUv2LSpTr+EQggnOXpULVHRqROc\nOAEbNqhBaD161OnlJKA4kKenJ/3792fDhg3OrorddOumJh4eO1Y1s5bx91cDV+67D06fdlr9hBA2\n2LdPTaFkChzbtsG779Z7ZVYJKA7WVBLz5v7yF+jSRc2+YiE5GUaPVmsKCyFczx9/qIn6rrgCIiJg\n92549VWIibHLy0tAcbCmlJg3MRjUHzNffw2ff17h4AsvwNq1VcyBL4Rwik2bYNQoNUlf9+5qedZn\nn4WW1lZMrzsZ2Ohgp0+fJjY2ltOnT+Pl5eWUOjjKpk1qXOOmTRAfb3ZgzRo1PuWPPyAoyEm1E8LN\naZpaFO+552DPHpX8vPtu1TxtIxnY6GJCQkKIjY0lPT3d2VWxuwED4IknYNw4KCoyO3DNNWrelv/7\nP6fVTQi3pWmwbJnqfXnvvXD77bB3rxqAXItgUhcSUBpAU+s+bO6RRyAkRPU4tPDyy6rr4TffOKVe\nQridkhI1rcXll6v/kI8+Cjt2qMn4fHwapArS5NUAPvjgA9asWcPChQtNlaG4uJiSkhKKi4sr7dvr\nWMXnpaWljB49mtjYWLt+vpMnoXdvtTrw0KFmB77+WvUkSU+HgAC7vqcQQldYCPPnq/xlRAQ8+SRc\nf71KdtZTbZu8JKA0gIyMDC677DI8PDzKvti9vLzw9PTEy8urbKvuuT3OPXfuHN9++y1ff/01Xbt2\ntetn/P57NYHkr79CVJTZgbvvhmbN1OIqQgj7KSiA996DV15R/fmnToXBg+0SSEwkoFjn1IACcOnS\nJQwGA56ennh4eJj+oRrc/Pnzefzxx1m+fDl9+/a162tPmwbr16sbk7IJlvPyVK+S+fPV3C1CiPrJ\ny1N/oL3+upoKfMoU6NfPIW8lSXkX5evri4+PD56enk4LJgApKSnMmjWL66+/nh8rLcdYP9OmwaVL\naiqgMsHB8M47avDK+fN2fT8h3Ep2tmrO6tABdu1SzQKff+6wYFIXjvxmawasBXwBH2AJMAUIBRYB\nccBB4DYgT79mCnA3UAL8A1itl/cF5uivuQIwrczhC8wD+gC5wBjgkJW6OP0OxdV8++23jBs3jnnz\n5nH99dfb7XUzM9Xv9//+pzqZlElJgdBQ9VeVoxUXq2kkjh5VW2Zm+b7p+bFjKq8THw9xcWoz7Zse\nW7RwfF2FqMmRI6pZa/58NUXFpEnQtm2DvLWrNXn5AwWAF/AT8H/AjUAO8BLwBBACTAa6Ah8B/YHW\nwLdAR0ADNgEP6I8rgDeAVcB9QHf9cQwwGhhrpR4SUKxYv349o0aN4j//+Q+33nqr3V532TI1+fCv\nv6oYAsCpU6rpa/FidZteV+fPVx0kTPsnT0J4OLRuXb7FxFjut2oF587BoUNw8KDlo2nf398ywFQM\nOsHB9fxJCVGNvXvV7f4XX6hc5KOPqt/bBuRqAcXEH3W3chfwOZAMZAFRQBrQBXV3UgqYGkxWAamo\nO47vgMv08rGAEbhXP2c6sBEVtI4D1oZ+SkCpwu+//87111/Ps88+y8SJE+32ug8/DIcPqzvysha+\nL75QA1d+/11Ne29O09SMp+aBwVrQuHTJepAwfx4VBd7e9fsApvpUDDbmjwZD9Xc44eF2TZCKJu7M\nGTUVyp49sHy5GiD8wAPqr7Oyv8waVm0DiqOHbnsAvwLtgVnAdiASFUzQH01rx0YD5rMoZqLuVIr0\nfZOjejn64xF9vxg4g2pSO2XPD9GU9erVi7S0NIYMGUJ+fj6P2GkerhdfVE1eb78N99+vF44eraYr\nvuMOdctuHjSOH4fmzSsHicREy+ehoQ3zJW0wqGkpWrZUK1NWpGkqOVoxyPz8c/nzixerDjbx8WrZ\n5Ea8PLSog6IiOHBABQ7TtmePejx3Ts3627kzJCSoHlyNrLu9owNKKXA5EAR8DVTs5qPpm8OlpqaW\n7RuNRoxGY0O8baPQqVMnfvzxR4YMGUJeXh6pqan17jjg66vGWCUmwpVXqrFWgFr68dVX1WjIvn3L\n7yyioyvftbgyg0F9hpAQsw9XwdmzlQPOr7+W7585owJL9+5q1lfT1q4deHo23GcR9qVpKoFuHixM\n26FD6ne+c2e19emjppro3Fn9H3DyHW1aWhppaWl1vr4ha/8UcAH4C6rJ6gTQCvge1eQ1WT/vBf3R\n1Jx1SD/H1OQ1DkgC/k55s9gGpMmr3rKysrjuuuswGo28+uqreNjhr+eFC+GZZ9R69JLjrqCgQE3S\nl55uueXkQNeulkGmRw81aE24joIClecwDxqmfS8vFSRMdxymrX179ddWI+FKOZRwVDNUHuCHukN5\nGrgO1SPrRVQQCcYyKT+A8qR8B9QdzEZUr69NwFdYJuV7oILLWGAUkpSvl9OnT3PDDTfQuXNn3nvv\nPbtMaHn33VBaCnPm1L9+biE/X02saR5ktm1T02dUDDLdujl8fia3VlqqellZa6LKzlZ3k+YBwxRA\nwsKcXXO7cKWA0gOYi8qjeADzgZdROY7FQCyVuw1PRXUbLkZ1Df5aLzd1G/ZD9fIyrTXrq79ub1SQ\nGqu/ZkUSUGrh/PnzjB49msDAQBYuXIhvPf+iOn9etW49+aTqPSzqQNNUrqni3czu3arJsGKg6dDB\ndZvNSkrUImynTkFurtpOnVJNhNZUbAay1ixU17KKzysGkIwMlberGDA6d1a5MFf9GduJKwUUVyIB\npZYuXbrEuHHjKCgo4PPPP6d58+b1er1t29QkxD//rP5PCjspKlLNLhUDTVaWWgWtRw/o2bM80ERG\n2q+dXtPU3VTFwFDd/qlT6pqgIPVFHRZW/hgQULluFf/fWvt/XNeyqr4TYmLKg0bHjm7dVisBxToJ\nKHVQXFzMxIkT2bdvH1999RVB9VzbZNYstTDXhg2Nqhm5cTp7FrZvt2wyS09XvcqsNZt5eFQfBKwF\nidOnVUcKU0AwDw7m+xXLgoOb/F/2TYUEFOskoNRRaWkpDz/8MD/99BOrVq0ioh6JYU2DW29VA9n7\n91ffYTVtBoNt59XluvBwaNPGjf4A1TTVPbvi3czOnep4VQGhquAQEtJg06IL55CAYp0ElHrQNI1p\n06bx6aef8u233xJTj/Wn8/LgjTfUEI3S0vJN0yyfV7fV5tyqzi8pUQPqjxxRd0tt2lS9xcSoCZOb\nLE1zendV4ZokoFgnAcUOXnnlFd566y1Wr15Nx44dnV0du9A01Ypz5EjV29Gjqsm/uqATHV3/wflC\nuBoJKNZJQLGT9957j9TUVFatWkWPHj2cXZ0GUVqqeoiaB5nDhy2fZ2erQfXVBZ2oKBkYLxoXCSjW\nSUCxo08++YSHHnqIJUuWMHDgQGdXxyUUF6sJjKu708nLU3P7mQeZ+HjVmahLF3VMWp6EK5GAYp0E\nFDv76quvuOuuu1i0aBFXX321s6vTKFy8qJrPzIOM+bROFy6owGLaTIGmQwfpFSecQwKKdRJQHGDt\n2rXceuutvP/++9x4443Ork6jd+qUCiy7dlluhw6puxnzYGMKOOHhzq61aMokoFgnAcVBNm/ezMiR\nI5k5cyZ33HGHs6vTJBUWqim/KgaaXbvUlFEVA02XLqopzQ6z5jhcaWkpp06dIisri/PnzxMYGEhQ\nUBBBQUH4+fk5dXVT4XrT14smrn///qxZs4brrruO/Px8/v73vzu7Sk2Oj095oDBnmtTWPMCsWaMe\nT5xQ00xZu6sJDHRsfUtKSjh58iRZWVllW3Z2tsVz05aTk0OLFi2IjIykRYsW5Ofnc+bMGc6cOUNx\ncTFBQUEEBweXBRnzzVp5xbJmTbq/d/U0TaO4uJiSkhJKSkqs7td0vLbcJfzLHYqD7d+/nyFDhnDP\nPfcwefLkmi8QDnXhgpqRpeIdze7daqC6KT9jCjKhoSpwmTZfX8vnBkMhubnZVQYG8+306dOEhIQQ\nGRlZtkVERFg8Ny/3qWJw5KVLl8qCS8UtLy/PpjKgVgEpKCgIg8FAcXFxrbeioqI6XVfV9bZ84Vd3\nXNM0PD098fT0xMvLq9K+tbKK+5s3bwZp8qpEAkoDOHr0KEOHDmXkyJE8//zz0lzhgkpL1ZpmplzN\ntm35/P57BqdPH+XChSwuXcqisDCLoiK1lZaqDc4CLTEYIvHwiMTLKwJv70h8fNTm56c2f/9I/P3D\n8fPzqjZAWXvu7W3fzctLo6joIvn5tgUk8yDk5eVVp83b27vO15pfb8sXfk3HPTw86v1/UHIo1klA\naSA5OTkMGzaM/v3789Zbb9llTRVRP4WFhezfv589e/ZYbLt37yY/P58OHToQExNj9Q7CtIWGhqJp\nHly6pHI6ps3ez4uK7LuVltY+EJkHPF9f2/bre66rTm0mAcU6CSgNKD8/n5EjRxITE8OcOXPwliHk\nDldaWkpmZmaloLFnzx4yMzNp06YNnTt3plOnThZbdHR0kw76paW1D0Lmge7SJct9a2W1OV7VuR4e\nle/cTPPRGQyW+1U92nJObc9dvlwCijUSUBrYhQsXuOWWW/Dy8mLRokVunRy1p9zcXKtBIyMjg+Dg\n4EoBo1OnTrRt27bKPIVwPk1Tc8uZB5zCQlVumoeu4n5Nj/Y658YbJaBYIwHFCQoLCxk/fjwnT57k\nyy+/JCAgwNlVahQKCgrIyMgoa5YyDxwlJSV06tSp0t1Ghw4d5Ocr7E6avKyTgOIkJSUl3HvvvaSn\np7NixQpCQ0OdXSWnKygoICcnh9zcXI4dO8bevXstgsbJkydp37691buNli1bSmcH0WAkoFgnAcWJ\nNE1j0qRJrF69mtWrVxMVFeXsKtmFpmkWwSEnJ8di31pZTk4OAOHh4YSHhxMZGUnHjh0t7jjatGmD\np6tmaYVbkYGNwuUYDAZefvllgoODGTx4MN988w3x8fHOrpYFTdM4f/58rYODh4dHWXAICwuzeOzS\npUvZvnm5v7+/sz+uEA4hAUU0CIPBwD//+U+CgoLopC8q7+HhgYeHR1mfeWvP67Nf07GzZ89aBAxP\nT0+LL37z/a5du1YKGGFhYRIchDAjTV6iwZWUlFBaWlr26Kz9gIAAi+Dg5+fn7B+NEC5FcijWSUAR\nQohaqm1AcfSIpjbA98B24A/gH3p5KPANsAdYDQSbXTMF2AvsAoaalfcF0vVjr5uV+wKL9PINQJy9\nP4QQQoiaOTqgFAGPAN2AgcD9wGXAZFRA6QSs0Z8DdAXG6I/DgLcpj46zgIlAR30bppdPBHL1steA\nF99WJEgAAAWxSURBVB35gRq7tLQ0Z1fBZcjPopz8LMrJz6LuHB1QTgBb9f1zwE6gNXAjMFcvnwuM\n0vdvAj5GBaKDQAZwBdAKCAA26efNM7vG/LU+B66x/8doOuQ/Szn5WZSTn0U5+VnUXUNO4hMP9AY2\nApFAll6epT8HiAYyza7JRAWgiuVH9XL0xyP6fjFwBtWkJoQQogE1VEBpgbp7eAg1D7Y5Td+EEEKI\nankDXwMPm5XtAkzDpVvpz0HlUsxXZ1qFavKKQjWXmYxD5VRM5wzU972Ak1bqkEF54JJNNtlkk822\nLQMXYkDlO16rUP4S8IS+Pxl4Qd/visq5+ABtgX2UJ+U3ooKLAVhBeVL+PsqDy1jgE7t+AiGEEC5h\nEFCKChK/6dswVI7jW6x3G56Kioq7gOvMyk3dhjOAN8zKfYHFlHcbjrf/xxBCCCGEEEIIOxiGutPZ\nS3kTm7uqapCpu/JE3TEvc3ZFnCwY+AyVo9xBeT7SHU1B/f9IBz5CtX64iw9QPW7TzcqqG4DudjxR\nzWPxqI4BW1GDKt1VFHC5vt8C2I17/zweBRYCS51dESebC9yt73sBQU6sizPFA/spDyKLgAlOq03D\nG4wa1mEeUF4CHtf3n6A81+2WElA9wEwq9iBzd1/ivoNAY1A5vKtw7zuUINSXqFB/je8GQlCBdRlw\nrVNr1PDisQwouygfIxhFeW/cKjXkwMaGZj7gEcoHSQrLQabu6DVgEqrDiDtri+pm/yHwK/Ae4K7z\n8Z8CZgKHgWNAHuqPDndW1QD0KjXlgKI5uwIuqgWqzfwh1HQ47mYEkI3Kn7jLbNtV8QL6oObM6wOc\nx33v4tujxsrFo2bmaAHc4cwKuRjTuJRqNeWAchSViDZpg+X0Le7IGzVjwQJUk5c7SkTN/3YANW/c\n1aixUu4oU982688/QwUWd9QPWIeaaLYY+B/qd8WdZWE5AD3biXVxOi/UwMh41EBJd0/KVzXI1J0l\n4945FIAfULN+A6TivrN190L1fvRD/V+Zi5od3Z3EUzkpb20Autu6HpVoy0B1CXRnVQ0ydWfJSC+v\nXqg7lN9Rf5W7ay8vUD2aTN2G56Lu6N3Fx6jcUSEq9/xnqh+ALoQQQgghhBBCCCGEEEIIIYQQQggh\nhBBCCCGEEEK4u1Jgvtlz05LStR38eBDVh7++59jqLuBNO72WEHXWlKdeEaK2zgPdgGb68yGoqUlq\nOy+cLefXZ665+v6/9arn9UJYJQFFCEsrgBv0/XGoEcSmSSRDUXOg/Q6sB3ro5WGokcR/oGbsNZ90\n8k7UrM6/Ae9Q8/+5ccA21Ght86kuzgGvoGY6SECNZN6tv7b5nFMtUXNybdI307FU1N3XT6hR4EII\nIRzoLCpIfIpaaOk3LOf7ehN4St+/Sj8O8AbwT31/OKrpLBQ1d9xS1GJvoGb1TdH3D1C5ySsaOIQK\nUJ7AGuAm/VgpcIu+38rsPG9UkHhDP/YRcKW+H4tahRFUQNmMe61CKBqY3PoKYSkdNUneOOCrCseu\nBP6k73+P+kIPQK12N1ovXwGcRt2lXAP0Bbbox/yAE9W8d3/9dXP15wuBJGAJUIKaKRrgigrnLaJ8\ngsdrsZwENQBojmpiWwpcqub9hagXCShCVLYU1byUjGpCMlfVGipVlc8Fptr4vlqF1zFQnmu5aLZf\n3XkGVMAptPL6BTbWQ4g6kRyKEJV9gGoi2l6h/EfKF10yonqAnUVNAX+7Xn49ahlZDdVkdQvlQSkU\n1QxVlc2oIGZq8hoLrLVy3ib9vFBUk9etZsdWA/8we96rmvcTQgjhIPlWysynuA8BvkAl5dcB3fXy\nUOBrVFL+XSzzI7ehci2/o5q+Bujl1nIooIKIKSn/fDV1u4vypPw7lOdQwoBP9PfbjsrbAEwHHrXy\nfkIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQjQ+/w9OFDBbZBGjsgAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x103454090>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pylab as plt\n",
    "%matplotlib inline\n",
    "order = np.arange(max_order+1)\n",
    "plt.plot(order,train_loss.mean(axis=0),'b-',label=\"Training loss\")\n",
    "plt.plot(order,cv_loss.mean(axis=0),'r-',label=\"CV loss\")\n",
    "plt.plot(order,ind_loss.mean(axis=0),'k',label=\"Independent test loss\")\n",
    "plt.legend()\n",
    "plt.xlabel('Model order')\n",
    "plt.ylabel('Mean squared loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>As expected, training loss (blue) always decreaes. The CV loss doesn't, and comes to a minimum at around 3rd order (this will not always be the case - it depends on the dataset generated at the start of the notebook). The independent loss typically has a more well-defined minimum.</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
